Prbatero/feat/data publishing local - #122
Conversation
…lease
Bucket A (additive) from the -sol audit split, Phase 1 (Local only; no PC):
- core/publishing/: base ABC + registry (PC lazy-loaded, disabled by default),
repository (per-dataset docs + optimistic concurrency), source (eligibility +
artifact resolution), lease (blob-lease coordinator), local_provider (immutable
copy to published/{datasetId}/).
- models/publishing.py; processors/publishing.py + assessment.py.
- Additive infra: config publishing block + PUBLISHED_DATASET type + publish
queue; data-layer load_bounded/load_page/update_locked; artifact-storage
copy/scoped-download/etag/size/exists/delete_prefix; blob.py max_bytes.
- hastefuncqueues publish trigger.
- Phase-1 unit tests (registry/repository/source/local_provider/lease): 45 pass.
Excludes Bucket B: reverted the high-risk load_all/load_all_from_partition
rewrite to main's versions; endpoint hardening + DTOs left for a separate
-hardening PR. See spec/features/data-publishing/sol-classification.md.
Bucket A only. Adds to api/hastefuncapi/function_app.py: publishing imports, active-caller/authorization helpers, the DeleteProject published-dataset guard, and 7 publishing routes (GetPublishingProviders, GetPublishDatasetOptions, GetPublishedDatasets, GetPublishedDataset, PutPublishDatasetQueueMessage, PutRetryPublishedDatasetQueueMessage, DeletePublishedDataset). All existing endpoints reverted to main's versions — the Bucket-B endpoint hardening (IDOR checks, strict *Request DTOs, server-managed-field enforcement, assessment size caps) is deferred to a separate -hardening PR. Tests: 19 publishing route tests + 6 queue-handler tests pass. Existing core suite: 191 pass (3 failures are pre-existing on main — clipBbox mocks).
Bucket A UI, gated behind the publishingEnabled flag: - New: PublishedDatasets section, PublishedDatasetRow, PublishDatasetModal (name/description prefill, asset checklist, provider-driven target dropdown), util/publishing.js, util/assessmentSummary.js (extracted from AssessmentReportModal). - Additive wiring: App/AppContext (publishingEnabled + provider fetch), AppBody route + AppSidebar nav, "Publish dataset..." in ModelResultsButton + EmbeddingModelRow, Database icon. - api.js: apiPut/apiDelete now accept any 2xx (async 202 queue responses) — minimal additive change; the shared-helper ApiError rewrite is NOT taken (deferred to -hardening). Existing 200 callers unaffected. Validated: full `vite build` compiles cleanly (all imports resolve). Vitest unit tests + config are preserved on -sol/archive and land in a follow-up UI-test-infra slice (needs vitest devDeps + package-lock regen).
UI (Published Datasets section): - Clickable Project/Layer cell -> opens the source project with that layer expanded (/project/:projectId/:imageLayerId). - Labeled "Actions" button (replacing the bare "...") and an "Actions" column header; keep the menu (download / open in Explorer / retry / unpublish). - New "View details" action -> modal with the generated description, project/ layer, model, target, status, publisher, headline assessment stats, and the published assets (kind + size). - "Published by" now mirrors the project Creator display: the publisher's login/email via limitTextLength, not the opaque object id. - Desktop column widths moved into a @media (min-width: 993px) rule scoped to .pgrid-page--published-datasets so the mobile card layout (inherited from .pgrid-page--model-catalog) is preserved; font URLs forced to https (CSP). Backend: - Capture the publisher's display name (client-principal userDetails) at publish time as PublishedDataset.publishedByName, so the UI can show a human name while the ownership check still uses the object id.
Add the data-publishing feature flag and queue name to the api/queues
Function App settings so the Local target runs in Azure:
- functions.bicep: PUBLISH_QUEUE_NAME ('publish-queue') and
PUBLISHING_ENABLED (from a new publishingEnabled param) in appConfigSettings.
- main.bicep / main.bicepparam: publishingEnabled param, default off,
sourced from HASTE_PUBLISHING_ENABLED.
The publish-queue and the publishing-locks blob container are auto-created at
runtime, and APIM operations are auto-synced from the deployed function app by
the postdeploy hook, so no queue/container/APIM resources are needed here.
Other Local knobs (PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES,
PUBLISHING_LOCK_CONTAINER) use code defaults.
Note: infra/main.json (compiled ARM artifact) is intentionally not regenerated
here to avoid a bicep CLI-version reformat; regenerate it with the repo's
pinned bicep version.
Add the publishing feature flag + queue/lock settings to the api and queues services so `docker compose up` exercises Local publishing against Azurite (the publish-queue and publishing-locks container auto-create at runtime). PC_PROVIDER_ENABLED stays false — no local GeoCatalog.
Drop the accidental "install" and "npm" entries from the UI dependencies (and their lockfile trees). Neither is imported anywhere; the npm CLI as a runtime dependency is what caused local image builds to fetch an npm tarball. Mirrors the earlier cleanup in #64 before they regressed back in.
There was a problem hiding this comment.
Pull request overview
Adds an asynchronous dataset-publishing workflow spanning the API, queue worker, storage layer, infrastructure, and UI.
Changes:
- Adds publishing models, persistence, locking, artifact handling, API routes, and queue triggers.
- Adds publish/catalog UI workflows with status polling and local downloads.
- Adds deployment configuration and backend tests.
Reviewed changes
Copilot reviewed 54 out of 57 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
ui/src/util/publishing.js |
Adds publishing status and artifact helpers. |
ui/src/util/icons.jsx |
Adds database icon support. |
ui/src/util/assessmentSummary.js |
Extracts assessment-summary formatting. |
ui/src/util/api.js |
Supports additional successful HTTP statuses. |
ui/src/Components/PublishedDatasets.jsx |
Adds the published-dataset catalog. |
ui/src/Components/PublishedDatasetRow.jsx |
Adds dataset details and actions. |
ui/src/Components/PublishDatasetModal.jsx |
Adds the publishing dialog. |
ui/src/Components/ProjectManagement/ModelResultsButton.jsx |
Adds publishing for model results. |
ui/src/Components/ProjectManagement/EmbeddingModelRow.jsx |
Adds publishing for embedding results. |
ui/src/Components/BuildingValidation/AssessmentReportModal.jsx |
Reuses the summary helper. |
ui/src/Components/AppSidebar.jsx |
Adds catalog navigation. |
ui/src/Components/AppBody.jsx |
Registers the catalog route. |
ui/src/assets/css/style.css |
Styles the dataset table and font URLs. |
ui/src/AppContext.jsx |
Adds publishing state. |
ui/src/App.jsx |
Loads publishing capabilities. |
ui/package.json |
Removes unnecessary install dependencies. |
infra/modules/functions.bicep |
Configures publishing for Function Apps. |
infra/main.bicepparam |
Reads the publishing feature flag. |
infra/main.bicep |
Passes publishing configuration. |
hastelib/tests/core/publishing/test_source.py |
Tests source resolution. |
hastelib/tests/core/publishing/test_repository.py |
Tests persistence and concurrency. |
hastelib/tests/core/publishing/test_registry.py |
Tests provider registration. |
hastelib/tests/core/publishing/test_local_provider.py |
Tests local publishing. |
hastelib/tests/core/publishing/test_lease.py |
Tests blob leases. |
hastelib/src/hastegeo/core/utils/blob.py |
Adds bounded streaming downloads. |
hastelib/src/hastegeo/core/publishing/source.py |
Resolves publishable artifacts. |
hastelib/src/hastegeo/core/publishing/repository.py |
Persists published datasets. |
hastelib/src/hastegeo/core/publishing/registry.py |
Registers publishing targets. |
hastelib/src/hastegeo/core/publishing/local_provider.py |
Implements local publishing. |
hastelib/src/hastegeo/core/publishing/lease.py |
Coordinates publishing operations. |
hastelib/src/hastegeo/core/publishing/base.py |
Defines the provider contract. |
hastelib/src/hastegeo/core/publishing/__init__.py |
Exports publishing primitives. |
hastelib/src/hastegeo/core/processors/metadata.py |
Adds bounded and paginated reads. |
hastelib/src/hastegeo/core/processors/assessment.py |
Generates bounded assessment reports. |
hastelib/src/hastegeo/core/models/publishing.py |
Defines publishing schemas and states. |
hastelib/src/hastegeo/core/data_layer/unified.py |
Exposes paginated storage operations. |
hastelib/src/hastegeo/core/data_layer/local_file_system_data_layer.py |
Adds bounded local reads. |
hastelib/src/hastegeo/core/data_layer/azure_postgresql_data_layer.py |
Adds bounded PostgreSQL reads. |
hastelib/src/hastegeo/core/data_layer/azure_data_lake_data_layer.py |
Adds bounded Data Lake reads. |
hastelib/src/hastegeo/core/data_layer/azure_cosmos_db_data_layer.py |
Adds bounded Cosmos reads. |
hastelib/src/hastegeo/core/data_layer/azure_blob_storage_data_layer.py |
Adds indexed blob pagination. |
hastelib/src/hastegeo/core/data_layer/abstract_data_layer.py |
Extends the data-layer contract. |
hastelib/src/hastegeo/core/artifact_storage/unified_artifact_storage.py |
Exposes publishing storage operations. |
hastelib/src/hastegeo/core/artifact_storage/local_file_system_artifact_storage.py |
Implements local artifact operations. |
hastelib/src/hastegeo/core/artifact_storage/azure_blob_artifact_storage.py |
Implements blob copy and scoped downloads. |
hastelib/src/hastegeo/core/artifact_storage/abstract_artifact_storage.py |
Extends the artifact-storage contract. |
docker/docker-compose.yml |
Enables publishing locally. |
api/hastefuncqueues/tests/test_publishing_handlers.py |
Tests publishing queue handlers. |
api/hastefuncqueues/tests/__init__.py |
Initializes the queue test package. |
api/hastefuncqueues/requirements.txt |
Updates publishing dependencies. |
api/hastefuncqueues/function_app.py |
Adds publishing queue triggers. |
api/hastefuncapi/tests/test_publishing_routes.py |
Tests publishing API routes. |
api/hastefuncapi/tests/__init__.py |
Initializes the API test package. |
api/hastefuncapi/function_app.py |
Adds publishing endpoints and authorization. |
Files not reviewed (1)
- ui/package-lock.json: Generated file
Suppressed comments (1)
ui/src/Components/PublishedDatasetRow.jsx:92
apiPutrepresents every 409 as the numeric value409, so thisawaitis treated as success. A retry rejected because of a stale/active operation silently refreshes the unchanged row instead of showing the existing “Retry failed” dialog.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
RC artifacts readyAll branch deployment references use the same RC tag:
|
… deploy paths
The Config drift check failed with:
Dead application settings (set but never read):
PUBLISHING_ENABLED (set by infra/modules/functions.bicep, read by no code)
That was a false positive. PUBLISHING_ENABLED is read in hastegeo.core.config
via _get_bool_env("PUBLISHING_ENABLED", False), but check_env_drift.py's AST
scanner only recognized os.getenv, os.environ.get, and os.environ[...]. The new
typed wrappers _get_bool_env and _get_bounded_int_env were invisible to it, so
no reader was found and the setting looked dead.
The blind spot covered five variables, not one: PC_PROVIDER_ENABLED,
PUBLISH_MAX_TOTAL_BYTES, PUBLISHED_DOWNLOAD_SAS_MINUTES and PC_VERIFY_ATTEMPTS
are read through the same helpers and were equally unseen. They simply did not
error because no deploy path emits them yet.
Teach the scanner to treat _get_*_env(NAME, ...) calls as reads. The wrappers
supply their own default in the signature, so such a read counts as optional
even when the call site passes no default.
Separately, fix the real drift the check could not see: PUBLISHING_ENABLED was
emitted by functions.bicep but not by deploy_apps.sh, so environments deployed
through the GitHub Actions path would never receive it and would silently fall
back to false while Bicep-provisioned environments got the flag. Emit it from
deploy_apps.sh too, defaulting off to match the Bicep param, and wire it
through deploy-apps.yml as a GitHub Environment variable (non-sensitive feature
flag, same treatment as VITE_SHOW_FOOTER).
|
Pushed The failure was a false positive. The blind spot covered five variables, not one — Plus one real issue the check couldn't see. To turn publishing on for an environment: I deliberately did not add it to All 13 checks green. Nothing else in the PR touched. |
RC artifacts readyAll branch deployment references use the same RC tag:
|
…s, polling, privacy Resolve Copilot review findings on the publishing feature: - Embedding eligibility: source resolver gated on inferenceStatus, but embedding models signal completion via `status`. Gate on `status` for modelType "embedding", else inferenceStatus — otherwise every embedding publish was rejected before the embedding artifact logic ran. - Publish action on EmbeddingModelRow used inferenceStatus (never set for embeddings); use the already-computed isProcessed. - Owner detection: populate appParams.identityId from the SWA principal object id so it matches PublishedDataset.publishedByUser; previously identityId was never assigned, so non-admin publishers were never recognized as owners and Retry/Unpublish were hidden. - Publish dialog reported "Publishing started" on a 409 conflict (apiPut returns the status rather than throwing); surface the conflict as an error instead. - Published Datasets polling captured a stale query (page/filters/search/sort); drive the interval through a ref to the latest fetch and pause it while the search is not ready. - Persist the publisher email/login rather than the display name (PublishedDataset.publishedByName) — display names resolve from Entra at read. - Remove aria-modal=false from the publish dialog (it is a focus-trapping modal). - Provider registry: a Planetary Computer target is "configured" once the GeoCatalog URL is set; do not also require an ingestion source (needed only for private containers), so the public-container mode is configurable.
RC artifacts readyAll branch deployment references use the same RC tag:
|
Publishing adds the queues app's first timer trigger (ReconcilePublishingOperations), which acquires a host-scoped Singleton lock. A colliding auto-generated host id (long app name / shared storage / deployment slots) then surfaces as "Unable to acquire Singleton lock" and NoScriptHost, leaving the app unhealthy. Document setting a unique AzureFunctionsWebHost__hostId per app/slot in the rollout operator notes. Applies to Local and PC (the timer ships with the Local feature).
…proxy
Published copies were written to `{hash(projectId)}/published/{datasetId}/<file>`
— a five-segment blob path (data/hash/published/datasetId/file). The UI already
rewrites download URLs through the managed-identity storage proxy
(get-artifacts) via toBrowserStorageUrl, but that proxy only accepts four
segments, so the rewrite returned null and the UI fell back to the raw blob SAS
URL. On the VNet-only storage account (defaultAction: Deny) a direct blob
request from the browser is rejected with AuthorizationFailure.
Flatten the published prefix to `published/{datasetId}/<file>` (three
in-container segments). datasetId is a UUID, so the project-hash level was
redundant for uniqueness. Downloads now match the get-artifacts proxy and are
served on-network via managed identity, like every other HASTE artifact
download.
Note: datasets published before this change keep their old five-segment path and
must be re-published to be downloadable.
RC artifacts readyAll branch deployment references use the same RC tag:
|
RC artifacts readyAll branch deployment references use the same RC tag:
|
The section fetched its first page with fetchDatasets(false), so no loading indicator showed and the page rendered blank (the component returns null while items === null) until data arrived — noticeable on slower cloud loads. Trigger the shared full-page loading overlay on the first load (items === null), matching the Model Catalog pattern; later filter/search/sort/page changes still refetch silently so the overlay doesn't flash on every interaction.
RC artifacts readyAll branch deployment references use the same RC tag:
|
…aining submit) The initial publishing commit moved the module-level `from hastegeo.core.processors.train import TrainPreprocessor` import into a local import inside PutCancelModelQueueMessage, but PutRunModelQueueMessage (and the retrain path) still reference TrainPreprocessor at module scope. That raised an unhandled NameError on every training submit, surfaced to the client as a 500 from PutRunModelQueueMessage. Restore the top-level import and drop the now redundant local one.
Description
This pull request introduces a new publishing queue processing system to the
hastefuncqueuesAPI, adds support for publishing operations in the artifact storage layer, and enhances local development/test support for publishing workflows. The main changes include new queue handlers for publishing, updates to the artifact storage abstraction, and supporting infrastructure and dependency updates.Publishing queue processing and handlers:
function_app.pyfor handling publishing queue messages, poison queue messages, and periodic reconciliation of publishing operations, utilizing the newPublishingProcessorandPublishQueueMessagetypes.test_publishing_handlers.pywith isolated async tests for the publishing queue handlers, including message decoding, error handling, poison queue logic, and reconciliation.Artifact storage abstraction and Azure Blob implementation:
AbstractArtifactStorageinterface with new methods for resolving artifact paths, copying artifacts, deleting by prefix, checking existence, getting size/etag, and generating scoped download URLs.AzureBlobArtifactStorageto implement the new interface methods, improved initialization and container creation logic, and fixed input validation instore_artifact. [1] [2] [3] [4] [5]Dependency and configuration updates:
azure-planetarycomputerandpystac[validation], and bumpedazure-coreversion. [1] [2]docker-compose.ymlto include new environment variables for publishing queue configuration and enabled publishing features for local development. [1] [2]Fixes #
Type of change
Checklist
cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)Testing
Additional context